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ABSTRACT 

The purpose of this study was to test a methodology that will 
help Santa Barbara City College (SBCC), California, to validate the course 
prerequisites that fall under the category of highest level of scrutiny — data 
collection and analysis--as defined by the Chancellor’s Office. This study 
gathered data for. the validation of prerequisites for three courses: (1) CHEM 

101, which requires completion of MATH 4 and ENG 70; (2) ECON 101, which 

requires completion of MATH 100; and (3) FILMST 101, which requires 
completion of ENG 80 and ENG 70. These three course were chosen in order to 
ensure there was a large enough number of students participating in the pilot 
for each of the selected courses. The study aimed to determine which of the 
two methods of instructors’ rating of students is most appropriate for use in 
the validation of prerequisites, and whether this process can be served by 
just one of the two suggested approaches: instructors assessment of student 
readiness or mid-term grades. For CHEM 101, correlation between the 
assessment .and readiness and mid-term grades was only 0.44, whereas for ECON 
101 and FILMST 101, the correlations were 0.727 and 0.854, respectively. The 
authors conclude that it would be sufficient to use just the mid-term grades 
for assessment, rather than requiring both measures. Student survey appended. 
(NB) 
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Executive Summary 



The purpose of this study was to test a methodology that will help SBCC validate the course pre- 
requisites that fall under the category of highest level of scrutiny - data collection and analysis - 
as defined by the Chancellor’s Office. This analysis is required for out-of-sequence 
communication and computation skills and non-course prerequisites. In Fall 2001, three courses 
were selected to participate in this pilot: CHEM 101, ECON 101 and FILMST 101. They had 
large enough enrollments to ensure a sample sufficient in size for conducting the test. 

The following information was collected from the participating sections: 

Student self reported completion of the required prerequisites prior to enrolling in the 
selected course either at SBCC or at other higher education institution. 

Instructor’s assessment of student readiness for the course. Instructors should have 
completed this assessment during the fourth or fifth week of the course. 

Instructor’s mid-term grades. Instructors should have assigned these grades during the 
eighth or ninth week of the semester. 

The Fall 2001 pilot helped identify some of the problems that the college will encounter in a 
systematic effort to validate the prerequisites that fall under the highest level of scrutiny as well 
as provided some indications regarding the adjustments that could be made when shifting from a 
pilot to a larger scale validation. 

The findings and recommendations from this study are as follows: 

1 . Although obvious, it is clear that without a reasonable number of students without the 
prerequisites for each course in this category, it will not be possible to draw strong 
conclusions. The primary problem encountered in this pilot was the low number of students 
who reported not completing the prerequisites. This has undermined to a large degree the 
ability to draw conclusions. It is recommended that either collect data for multiple semesters 
or collect data from all sections offered in a given semester for each course to be analyzed 
emphasizing the need that all students present fill in the prerequisite surveys. 

2. It is possible that for courses with below college level prerequisites even if data from 
multiple semesters or all sections offered in one semester are collected, the number of 
students without prerequisites will still be too low. It is recommended that the college 
strongly consider the need for entering the information regarding courses completed at other 
institutions for all students who enroll. Having the actual courses completed rather than self 
reported data would not only help in the prerequisite validation process but in many other 
college business processes such as counseling, preparation of student educational plans and 
degree audit. In the interim, the self reported information is the only avenue available. Before 
proceeding with the large scale data collection and analysis, it would be helpful to conduct 
one more pilot with two or three courses that have higher level prerequisites. 

3. Colleges are allowed to combine various analysis approaches. It seems that the four-cell 
analysis of mid-term grades is better than the analysis of final grades. Thus, it is 
recommended that this approach be utilized, if the college will proceed with the validation 
process for all courses in this category. 

4. Although for one of the three courses the correlation between the instructor’s assessment 
of student readiness and mid-term grades was lower than desired, it is safe to assume that it 
would be sufficient to use just the mid-term grades rather than requiring both measures. 
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Introduction 



The purpose of this study was to test a methodology that will help SBCC validate the course pre- 
requisites that fall under the category of highest level of scrutiny - data collection and analysis - 
as defined by the Chancellor’s Office. This analysis is required for out-of-sequence 
communication and computation skills and non-course prerequisites. “The basic premise is that 
the college must demonstrate, using sound research practices, that students are highly unlikely to 
succeed without these skills” (The Academic Senate for California Community Colleges, 1997, 
P-4). 

The Office of Institutional Assessment, Research and Planning conducted an analysis in May 
2000 for all SBCC courses that fall in this category using the distribution of final grades over a 
three-year period (from Fall 1996 to Fall 1999). That analysis did not help in making the case, 
for most courses, that the prerequisites influence the rate of success of those who met them as 
opposed to those who did not. One major problem in the data available in the college student 
information system is that courses completed at other colleges or universities are not recorded, 
thus the information on actual completion of pre-requisites is just partial, based only on 
completion of such courses at SBCC or placement tests taken at the college. This pilot attempted 
to address this deficiency in data available. Another issue was that final grades (excluding Ws) 
represent only those students who chose to stay and complete the course. After the exclusion of 
Ws, the percentage of failing grades is, generally, very low, thus limiting the validity of the 
analysis. 



Pilot Methodology 

Using the suggestions in the Academic Senate’s (1997) document, the following instruments 
were used to gather data for the validation of prerequisites for three courses (CHEM 101, ECON 
101 and FILMST 101) in Fall 2001: 

- Student self reported completion of the required prerequisites prior to enrolling in the 
selected course either at SBCC or at other higher education institution (see Appendix 1 
for example of student questionnaire). This questioimaire was administered at the 
begiiming of the semester during a class session. 

Instructor’s assessment of student readiness for the course (see Appendix 2 for example 
of questioimaire). Instructors should have completed this assessment during the fourth or 
fifth week of the course. 

- Instructor’s mid-term grades. Instructors should have assigned these grades during the 
eighth or ninth week of the semester. 

The instruments and process were developed in collaboration with the Chair of the Curriculum 
Committee and presented to the committee for approval. 

The main reason for choosing CHEM 101, ECON 101 and FILMST 101 for testing the 
methodology was the need to ensure that a large enough number of students participated in the 
pilot for each of the selected courses. 
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Based on the information included in the 200 1 -02 College Catalog, the three courses included in 
the pilot have the following prerequisites: 

CHEM 101 requires completion of MATH 4 (or eligibility for MATH 100) and completion of 
ENG 70 (or eligibility for ENG 103) 

ECON 101 requires completion of MATH 100 (or eligibility for MATH 107) 

FILMST 101 requires completion of ENG 80 (or eligibility for ENG 100) and completion of 
ENG 70 (or eligibility for ENG 103). 

The data were analyzed using the model suggested by the Chancellor’s Office (the Four Cell 
Process, described in the next section, using the data collected for the three courses selected) 
augmented by additional tests deemed appropriate (such as comparisons between distribution of 
mid term grades and actual final grades - excluding Ws - for the courses selected and 
correlations between instructor’s assessment of student readiness and mid-term grades). 



Analysis and Results 



Student Responses to Prerequisite Ouestioimaire 

The key component in the validation of course prerequisites is having sufficient students in each 
course who have not completed them such that a comparison can be conducted between the 
course success of those with and those without prerequisites. Based on the self-reported 
information by students in the three courses, of the students who responded, most of them 
indicated having completed the required prerequisites (see Table 1, 2 and 3). Due to this 
distribution of responses, the essential statistical component of such a study (chi-square between 
the success of those who succeeded with prerequisite and those without) could not be calculated. 
This situation can have at least three explanations: 1) it might be that, generally, students actually 
follow the published requirements regarding courses and enroll having completed the 
prerequisites; 2) these courses have pre-requisites below the college level for both English and 
Mathematics, thus it is actually expected that most students should have reached these levels of 
preparation before enrolling; or 3) given that these surveys were not anonymous, there might be 
some degree of inflation in the responses. 



Table 1. CHEM 101 Student Responses to Prerequisite Survey 




Eligible for MATH 100 


Eligible for ENG 103 


Response 


N 


% 


N 


% 


Yes 


77 


82% 


81 


86% 


No 


4 


4% 


1 


1% 


No Response 


13 


14% 


12 


13% 


Total 


94 




94 
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Table 2. ECON 101 Student Responses to Prerequisite Survey 




Eligible for MATH 107 


Response 


N 


% 


Yes 


97 


66% 


No 


11 


8% 


No Response 


38 


26% 


Total 


146 





Table 3. FILMST 101 Student Responses to Prerequisite Survey 




Eligible for ENG 100 


Eligible for ENG 103 


Response 


N 


% 


N 


% 


Yes 


109 


81% 


106 


79% 


No 


7 


5% 


12 


9% 


No Response 


19 


14% 


17 


13% 


Total 


135 




135 





Instructors’ Assessment of Student Readiness and Mid-term Grades 



One of the purposes of this pilot was to determine which of the two methods of instructor’s 
rating of students is most appropriate for use in the validation of prerequisites and whether this 
process can be served by using just one of the two suggested approaches: instructor’s assessment 
of student readiness or mid-term grades. The frequencies for the two approaches are listed in 
Tables 4 through 9. For CHEM 101 the correlation between the assessment of readiness and 
midterm grades was only 0.44 (significant at <.0005) whereas for ECON 101 and FIMLST 101, 
the correlations were very high, 0.727 and 0.854, respectively (both significant at <.0005). This 
indicates that for the latter two courses, using just one of the methods is sufficient. For CHEM 
101, the somewhat low correlation prevents drawing a conclusion in this regard. For this course, 
the assessment of student readiness was, generally, much better than the mid-term grades, 
whereas for the other two courses the readiness assessment and the mid-term grades were, 
generally, close. 

For FILMST 101 the distribution of the mid-term grades matches very closely the distribution of 
the actual final grades given by the instructor who participated in the pilot in Fall 1999 and 2000. 
However, for the other two courses the pilot mid-term grades are either much lower than the 
final grades assigned by the instructor in the past or much higher. 
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Table 4. CHEM 101 Instructor's Assessment of Student Readiness 


Assessment 


N 


% 


Very prepared. The student will definitely be successful. 


29 


37% 


Prepared above average. 


25 


32% 


Prepared. With sufficient study, the student could be 
successful in this class. 


13 


16% 


Somewhat prepared but below the level needed to 
succeed in this class. 


4 


5% 


Not at all prepared. The student will have difficulty in 
this class. 


8 


10% 


Total Students Assessed 


79 




Students not assessed because they dropped 


15 




Total Students 


94 





Table 5. CHEM 101 Comparison of Pilot Midterm Grades to Actual Final Grades for the Last Two Fall Semesters 




Pilot Mid-Term Grades 
Fall 2001 


Actual Final Grades All 
CHEM 101 Sections Fall 
1999 & 2000 


Actual Final Grades for Sections 
Taught by G. Carroll Fall 1997 & 1998 
(Instructor did not teach CHEM 101 in 
Fall 1999 or 2000) 


Grade 


N 


% 


N 


% 


N 


% 


A 


11 


14% 


37 


14% 


23 


19% 


B 


12 


15% 


72 


28% 


33 


28% 


C 


23 


29% 


96 


37% 


37 


31% 


D 


22 


28% 


17 


7% 


6 


5% 


F 


11 


14% 


35 


14% 


19 


16% 


Total Grades 


79 




257 




118 




No Grades 


15 












Total 


94 













Table 6. ECON 101 Instructor's Assessment of Student Readiness 


Assessment 


N 


% 


Very prepared. The student will definitely be successful. 


5 


4% 


Prepared above average. 


23 


19% 


Prepared. With sufficient study, the student could be successful 
in this class. 


69 


57% 


Somewhat prepared but below the level needed to succeed in 
this class. 


23 


19% 


Not at all prepared. The student will have difficulty in this 
class. 


2 


2% 


Total Students Assessed 


122 




Students not assessed because they dropped 


24 




Total Students 


146 
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Table 7. ECON 101 Comparison of Pilot Midterm Grades to Actual Final Grades for Last Two Fall Semesters 




Pilot Mid-term Grades 
Fall 2001 


Actual Final Grades All ECON 
101 Sections Fall 1999 & 2000 


Actual Final Grades for Sections 
Taught by C. Barr Fall 1999 & 
2000 


Grade 


N 


% 


N 


% 


N 


% 


A 


27 


21% 


62 


16% 


20 


10% 


B 


28 


21% 


104 


26% 


36 


17% 


C 


35 


27% 


143 


36% 


87 


42% 


D 


21 


16% 


47 


12% 


37 


18% 


F 


20 


15% 


43 


11% 


26 


13% 


Total Grades 


131 




399 




206 




No Grades 


15 












Total 


146 













Table 8. FILMST 101 Instructor's Assessment of Student Readiness 


Assessment 


N 


% 


Very prepared. The student will definitely be successful. 


22 


19% 


Prepared above average. 


35 


30% 


Prepared. With sufficient study, the student could be successful 
in this class. 


34 


29% 


Somewhat prepared but below the level needed to succeed in 
this class. 


18 


16% 


Not at all prepared. The student will have difficulty in this class. 


7 


6% 


Total Students Assessed 


116 




Students not assessed because they dropped 


19 




Total Students 


135 





Table 9. FILMST 1 01 Comparison of Midterm Grades to Final Grades for Last Two Fall Semesters 




Pilot Mid-Term Grades Fall 2001 


Actual Final Grades All 
FILMST 101 Sections 
Fall 1999 & 2000 


Actual Final Grades for 
Sections Taught by M. 
Perona Fall 1999 & 2000 


Grade 


N 


% 


N 


% 


N 


% 


A 


11 


9% 


46 


14% 


15 


oo 


B 


28 


24% 


108 


32% 


44 


24% 


C 


45 


38% 


114 


34% 


71 


39% 


D 


24 


21% 


45 


13% 


32 


18% 


F 


9 


8% 


26 


8% 


18 


10% 


Total Grades 


117 




339 




180 




No Grades 


18 












Total 


135 
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Student Performance and Four-Cell Analysis 



Although the low number of students without prerequisites met indicated that the analysis would 
not yield relevant results, in order to test all analytical options suggested in the Academic 
Senate’s document, a four-cell process was attempted for the three courses. The four-cell 
analysis has the following purposes: 

to test whether the premise that being successful in a course and having completed the 
required prerequisites are independent of each other is correct (this is done by calculating 
a Pearson chi-square statistic). In other words, this test serves the purpose to show that 
having met the prerequisite does make a difference in being successful in the course. 

- to determine the right/wrong ratio. A case is “right” if the student was successful and met 
the prerequisite or if a student without the prerequisite failed. A case is “wrong” if a 
student with the prerequisite failed or a student without the prerequisite succeeded. The 
minimum desired level for this ratio is 2. 

- to determine the incremental gain in success which represents the percentage gain 
between the percent of successful students when all students are included versus the rate 
of successful students when only those with the prerequisites are included. Generally, the 
incremental gain should be at least 10% to make the case that the prerequisites make a 
difference in student success. 

As mentioned at the outset, because for all three courses there were too few students without the 
prerequisite(s), the chi-square statistic could not be calculated. 

For CHEM 101, both the right/wrong ratio as well as the incremental gain in success was below 
the minimum desired levels (see Table 10). Again, this is a direct consequence of not having 
enough students without the prerequisite. For ECON 101 (see Table 1 1), the right/wrong ratio 
was 3.2 (higher than the minimum level of 2) but the incremental gain in success was low. For 
FILMST 101 the ratios were also well above the level of 2 but the incremental gains were low. 
Although some of the results of this pilot would be helpful, overall it seems they are not strong 
enough for any of the three courses to justify their pre-requisites. The primary cause for not 
being able to draw more forceful conclusions was the low number of students without the 
prerequisites. 
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Table 10. CHEM 101 Midterm Grade vs Prerequisite Completion 




Eligible for 
Math 100 










Midterm Grade 


Yes 


No 


Total 


Chi-Square 


RightAVrong 

Ratio 


Incremental Gain in 
Success 


A, B, or C 


43 


1 


44 


Not enough students 
without prerequisite to 
calculate 


1.53 


0.02 


D or F 


29 


3 


32 




46:30=1.53 (desired 
level 2.0) 


before applying the 
prerequisite 
44/76=0.58 


Total 


72 


4 


76 






after applying the 
prerequisite 43/72=0.6 














2% gain 


A,B,orC 


60% 


25% 


58% 








D or F 


40% 


75% 


42% 
























Eligible for 
ENG 103 










Midterm Grade 


Yes 


No 


Total 


Chi-Square 


RightAVrong 

Ratio 


Incremental Gain in 
Success 


A, B, or C 


45 


0 


45 


Not enough students 
without prerequisite to 
calculate 


1.48 


0.01 


D or F 


31 


1 


32 




46:31=1.48 (desired 
level 2.0) 


before applying the 
prerequisite 
45/77=0.58 


Total 


76 


1 


77 






after applying the 
prerequisite 
45/76=0.59 














1 % gain 


A, B, or C 


59% 


0% 


58% 








D or F 


41% 


100% 


42% 
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Table 11. ECON 101 Midterm Grade vs Prerequisite Completion 





Eligible for 
Math 107 










Midterm Grade 


Yes 


No 


Total 


Chi-Square 


RightAV rong 
Ratio 


Incremental Gain in Success 


A, B, or C 


76 


7 


83 


Not enough 
students without 
prerequisite to 
calculate 


3.20 


0.02 


D or F 


18 


4 


22 




80:25=3.20 
(desired level 2.0) 


before applying the prerequisite 
83/105=0.79 


Total 


94 


11 


105 






after applying the prerequisite 
76/94=0.81 














2% gain 


A, B, or C 


81% 


64% 


79% 








D or F 


19% 


36% 


21% 









Table 12. FILMST 101 Midterm Grade vs Prerequisite Completion 




Eligible for I 
ENG 100 1 








Midterm Grade 


Yes 


No 


Total 


Chi-Square 


Right/Wrong Ratio 


Incremental Gain in 
Success 


A, B, or C 


78 


2 


80 


Not enough 
students without 
prerequisite to 
calculate 


3.19 


0.03 


D or F 


24 


5 


29 




83:26=3.19 (desired 
level 2.0) 


before applying the 
prerequisite 80/109=0.73 


Total 


102 


7 


109 






after applying the 
prerequisite 78/102=0.76 














3% gain 


A, B, or C 


76% 


29% 


73% 








D or F 


24% 


71% 


27% 
























Eligible for 
ENG 103 










Midterm Grade 


Yes 


No 


Total 


Chi-Square 


Right/Wrong Ratio 


Incremental Gain in 
Success 


A, B, or C 


76 


5 


81 


Not enough 
students without 
prerequisite to 
calculate 


2.96 


0.04 


D or F 


23 


7 


30 




83:28=2.96 (desired 
level 2.0) 


before applying the 
prerequisite 81/11 1=0.73 


Total 


99 


12 


111 






after applying the 
prerequisite 76/99=0.77 














4% gain 


A, B, or C 


77% 


42% 


73% 








D or F 


23% 


58% 


27% 
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Conclusions and Recommendations 



The Fall 2001 pilot helped identify some of the problems that the college will encounter in a 
systematic effort to validate the prerequisites that fall under the highest level of scrutiny as well 
as provided some indications regarding the adjustments that could be made when shifting from a 
pilot to a larger scale validation. 

The findings and recommendations from this study are as follows: 

1 . Although obvious, it is clear that without a reasonable number of students without the 
prerequisites for each course in this category, it will not be possible to draw strong 
conclusions. One option is to collect data for multiple semesters. Another option is to 
collect data from all sections offered for a given course in one semester and emphasize 
the need that all students present during survey administration fill in the information. 

2. It is possible that for courses with below college level prerequisites even if data from 
multiple semesters are collected, the number of students without prerequisites will still be 
too low to conduct an analysis. It is recommended that the college strongly consider the 
need for entering the information regarding courses completed at other institutions for all 
students who enroll. This is customary practice at many institutions. Having the actual 
courses completed rather than self reported student data would not only help in the 
prerequisite validation process but in many other college business processes such as 
counseling, preparation of student educational plans and degree audit. In the interim, the 
self reported information is the only avenue available. Before proceeding with the large 
scale data collection and analysis, it would be helpful to conduct one more pilot with two 
or three courses that have higher level prerequisites. 

3. Colleges are allowed to combine various analysis approaches. It seems that the four cell 
analysis of mid-term grades is better than the analysis of final grades. Thus, it is 
recommended that this approach be utilized, if the college will proceed with the 
validation process for all courses in this category. 

4. Although for one of the three courses the correlation between the instructor’s assessment 
of student readiness and mid-term grades was lower than desired, it is safe to assume that 
it would be sufficient to use just the mid-term grades rather than requiring both measures. 



References: 
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Appendix 1. STUDENT SURVEY FALL 2001 

FILM STUDIES 101, SECTION 



Student Name: «Name» 

Survey Code: «Survey_Code» 

Please fill in EACH CIRCLE COMPLETELY like this • DO NOT check or circle like this 

0 0 ® 

Please answer the two questions below: 

1 . During your studies at SBCC or at any other college or university (high school courses 
should not be considered) have you: 

• Completed English 80 Effective Writing Techniques, OR 

• Completed an English class covering writing competency and fluency, OR 

• Completed a more advanced English class than those described above (for example. College 
Fundamentals of Composition; College Composition and Reading), OR 

• Scored on an English placement exam that you took when you started (SBCC or any other 
college) at a level higher than English 80 or its equivalent (for example, a score that allowed 
you to enroll directly in ENG 100 - Fundamentals of Composition). 

If ANY of the above applies to you, then please fill in the circle for YES (below). Otherwise, 
please fill in the circle for NO. 

O Yes 

O No 



2. During your studies at SBCC or at any other college or university (high school courses 
should not be considered) have you: 

• Completed English 70 Effective Reading & Study Skills Techniques, OR 

• Completed an English class covering reading, study skills, and vocabulary development, OR 

• Completed a more advanced English class than those described above (for example. College 
Composition and Reading; College Writing), OR 

• Scored on an English placement exam that you took when you started (SBCC or any other 
college) at a level higher than English 70 or its equivalent (for example, a score that allowed 
you to enroll in ENG 103 - Improvement of College Reading). 

If ANY of the above applies to you, then please fill in the circle for YES (below). Otherwise, 
please fill in the circle for No. 



o 

o 



Yes 

No 



Appendix 2. Fall 2001 Student Readiness Questionnaire 



Instructor: Course: FILMST 101 Section: 

Date Provided: 

Date Returned: 

Please fill in EACH CIRCLE COMPLETELY like this • DO NOT check or circle like this 

0 0 ® 





Assessment of Student Readiness for the Class 


Student 

Name 


SSN 


Not at all 
prepared. 
The student 
will have 
difficulty in 
this class. 


Somewhat 
prepared but 
below the level 
needed to succeed 
in this class. 


Prepared. With 
sufficient study, 
the student could 
be successful in 
this class. 


Prepared 

above 

average. 


Very 

prepared. The 
student will 
definitely be 
successful. 






o 


o 


o 


o 


O 






o 


o 


o 


o 


o 






o 


o 


o 


o 


o 
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Appendix 3. Fall 2001 Mid-Term Grades 



Instructor: Course: FILMST loi Section: 

Date Provided: 

Date Returned: 

Please fdl in EACH CIRCLE COMPLETELY like this • DO NOT check or circle like this 

0 0 ® 





Mid-Term Grade 


Student 

Name 


SSN 


F 


D 


C 


B 


A 






o 


o 


o 


o 


o 






o 


o 


o 


o 


o 
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